There are many cases with image data where it is desirable to combine or segment images. These may include the segmentation of images into several or many ‘tiles’, for processing of each image, or the combination of several source images together on one display. There are even cases where it is desirable to segment very large images, process parts of the image through separate processing platforms, and to either combine them into a displayable stream for display on one very high resolution display, or to drive many physically adjacent displays that give the illusion of being one display. This application relates to the combining, segmentation, and separate processing of sections of these images.
Consider first the process of producing finished cinematographic works (films). For the first Eighty years in the history of film production, all work in editing, special effects, and compositing was done optically. This involved manually cutting film, and optically printing from one or more pieces of film onto a new film stock.
Approximately twenty years ago, it was demonstrated that the film as exposed in a film camera could be digitised in a frame by frame basis, and the result of this digitisation stored in digital storage media (such as magnetic computer discs). This data could then be processed by use of a digital computer, and a new digital image created from the first. Typical effects to be performed on the digital image may be to increase or decrease the brightness of a scene, or to alter one colour of the image differentially to other colours of that image. Other effects include the combination of two or more different images into one composite scene (this process is referred to as compositing). The resulting digital images may be shot back to film, using a film recorder. Thus this process has created a finished film from imperfect elements. This process is now common, and is referred to as ‘Digital Intermediate’, as the intermediate results only occur in the digital domain.
It will be appreciated that it is now possible to film digitally, avoiding the need to perform a separate step of digitising the images. Equally, a digital projector may display the digital images directly, thereby removing the need to transfer the digital images back to film.
The images and/or graphics are typically stored as a 2-Dimensional Matrix of Digital values, as illustrated in FIG. 1. To display images in this form, it is usual to have a Digital frame store 1, input means 2, and an image display means 3. It is usual in this arrangement to have a Digital-to-Analogue converter 4 which converts the digital stored values into equivalent analogue values for driving display means such as a cathode ray tube display.
The processing of digital images takes place in a variety of systems. Since a complete movie may be represented digitally by many Terabytes of data, significant computational resources are required to perform the necessary processing in a cost effective manner. Dedicated hardware systems such as the ‘iQ’ are available from the UK organisation Quantel Ltd., based in Newbury, Berkshire, UK. The dedicated hardware offers the necessary high speed performance, but has little flexibility for the addition of new features. Also, because the market for such systems is relatively very small compared with general purpose computers such as Personal Computers (PCs), the cost is relatively high. Thus there are several alternative systems available that use a general purpose PC. One such system is the ‘Lustre’ system from the Canadian company Discreet Logic. These systems have a lower cost, as the hardware is a mass commodity PC. However, since PCs are built for a wide variety of purposes, such systems do not perform optimally with very large quantities of data as is found in digitised film.
However, since PCs are built for a wide variety of purposes, such systems do not perform optimally with very large quantities of data as is found in digitised film. Thus the operator of such systems often has to wait whilst the system implements his required changes. This means that interactive response, where the operator alters a variable control, and wants to see interactively how much change is made whilst he alters the control cannot happen. This results in a great amount of frustration and lost productivity.
The Applicant has recognized that systems containing a number of industry standard PC computers to process different parts of the same film frame can be extremely useful. At least in preferred embodiments, these systems can result in much greater efficiency and provide real-time response in nearly all circumstances.
One problem occurs where it is necessary to show the changes made by each computer in combination, to show a desired change on the whole image. Whilst it would be quite usual in a multi PC architecture for each PC to write into part of a frame buffer, and to assemble the combined image in that buffer, for display to a display device such as a Monitor or projector, in practice this frame store would be expensive and cumbersome. Further disadvantages of using a frame store include the added delay that this causes. Thus frames viewed are yet another frame later than the operator's controls. Thus by the time the operator sees a frame, the computer is already handling frames that are ahead of this displayed frame. This lag makes the Human computer interaction difficult. The Applicant's co-pending UK patent application GB 0414897.9 filed on 2 Jul. 2004 teaches how problems such as these may be overcome using standard PC graphics cards, incorporating a simple modification to synchronise the output of the graphics cards, together with a low cost combiner card for combining the images from the graphics cards.
There are two principal classes of visual operation that are desirable in an image workstation. Firstly there are the manipulations that are performed on the data. Such operations include the lightening or darkening of an image, or part of an image. Other changes of this class may include the altering of the hue, saturation, or ‘green-ness’ of an image. A third example in this class may be to improve the sharpness of the image or part of the image. The defining factor of this class of change is that it is desired to change the look of the image or images for output. In other words, these are the changes that form part of the purpose and function of the ‘Digital Intermediate’ workflow.
A second class of function exists, where the function is only for display. One such function may be to make the image larger, so that a cut-out of a feature to be guided by the operator can be performed more easily. In this case, the enlargement is for the convenience of the operator to work on the image—there is no desire to make the output image larger. In another similar mode in this class, it may be desirable to compare two images. In digital film post production, it is often desirable to make changes on an image, whilst comparing the original (unchanged) image with the image that is being changed. This is often necessary to show how great or small a change has been made, as it is extremely difficult for the operator to remember precisely details of the original image, such as colour. FIG. 2 shows the monitor display that is desirable for the operator to perceive differences between two images. A first image 5 is displayed on the left hand side and a second image 7 is displayed on the right hand side. The second image 7 is the modified version of the first image 5. This class of display operation is intended to make the operator's job easier, and not intended to change the final film output to show both images. Putting it another way, this class of operation is intended to indirectly enable the operator to use the first class of operation to perform Digital Intermediate functions.
Some operations, such as a resize, can come into either class of operation. In the first class, it is desired to permanently change the size of the output image, to frame the material in a different way. In the second, it is desired not to change the output image, but to merely display it larger, so that the operator can see an area of the picture in greater detail.
To meet this need, it is desirable in the system described in the Applicant's co-pending application GB 0414897.9 to write onto the overall composite monitor space two different images. Thus it is necessary to allow the image combiner, described in the Applicant's co-pending application, to enable the display of two different images, rather than the display of different parts of the same image. This basically extends the concept introduced in GB 0414897.9 to allow display of a composite image from multiple graphics cards, without needing a frame buffer. The extensions consist of methods of using these same building blocks to display more complex image fields that meet the needs of post production, but by techniques that still do not necessitate a complex and expensive frame buffer.
There are a range of further processing techniques commonly employed when handling digital images. For example, there are many cases where it is desirable to segment images and display a number of these segments simultaneously, often but not limited to ‘video Wall’ displays. These are often the preferred image display means to overcome the physical limitations in display technologies such as flat panel displays and cathode ray tube displays. FIG. 3 illustrates an arrangement whereby six display panels 9, arranged in a 3×2 array, are used to display a large image at a size that it is not possible for any one display to be physically used. In this architecture it is necessary to segment an image into tiles, each of which is individually loaded into a digital frame store for that tile, and each tile is individually displayed. It is only the physical proximity of the display panels that creates the illusion of one image being displayed.
A system 11 for splitting an image into six tiles for display on a video wall is shown schematically in FIG. 4. The system 11 comprises an image disc store 13 and an image splitter 15 for splitting the image data into six separate blocks of data corresponding to the image tiles. The blocks of data are supplied to respective frame stores FS1 to FS6 and the image tiles displayed on corresponding screens 17, 19, 21, 23, 25, 27.
For many years there have existed a class of device for professional use known as ‘Video Switchers’. These devices have the properties of being able to switch between multiple streams of video information. These devices have further capabilities of being able to take various parts of several images and create a composite image. These devices are also referred to as DVE (Digital Video Effects) systems. These have been manufactured and sold for many years by companies such as PINNACLE Systems, based in Mountain View, Calif., USA. Typical usage of such systems is to provide dynamic transitions between image sequences, where the first image wipes into a second, or where the first image undergoes a ‘page turn’ to reveal the second image on the ‘reverse side’ of that image.